1. Field of the Invention
The present invention relates to an image processing technology which reduces a blur component contained in an image by an image restoration process.
2. Description of the Related Art
An image obtained by imaging an object using an image pickup apparatus such as a digital camera contains a blur component as an image deteriorating component which is caused by a spherical aberration, comatic aberration, a field curvature, or astigmatism of an image pickup optical system (hereinafter, referred to simply as an “optical system”), or the like. Such a blur component is generated by forming an image with a certain spread although a light beam emitted from one point of an object is ideally focused on one point on an imaging plane again if there is no influence of diffraction without any aberrations.
The blur component is optically represented by a point spread function (PSF), and is different from a blur caused by the defocusing. A color image may run due to a difference of the way of the blur for each light wavelength if it is caused by an on-axis chromatic aberration, a spherical aberration of a color, or a comatic aberration of a color.
As a method for correcting a blur component of an image, a correction method using information of an optical transfer function (OTF) of an optical system has been known. The method is called an image restoration or an image recovery, and hereinafter a process of correcting (reducing) the blur component of an image using the information of the optical transfer function (OTF) of the optical system is referred to as an image restoration process.
A summary of the image restoration process is as follows. A deteriorated image containing a blur component (input image) is defined as g(x,y), and an original image, which is not deteriorated, is defined as f(x,y). A point spread function (PSF) that is a Fourier pair of the optical transfer function is defined as h(x,y). In this case, the following expression is satisfied. In the expression, symbol * denotes a convolution, and symbol (x,y) denotes a coordinate on an image.g(x,y)=h(x,y)*f(x,y)
When the above expression is converted into a display format on a two-dimensional frequency plane by a Fourier transform, as described in the following expression, it is changed to a format of product for each frequency. Symbol H denotes a value which is obtained by performing the Fourier transform of the point spread function (PSF), and it is the optical transfer function (OTF). Symbol (u,v) denotes a coordinate on the two-dimensional frequency plane, i.e. frequency.G(u,v)=H(u,v)·F(u,v)
In order to obtain the original image from the deteriorated image, as described below, both sides are divided by H.G(u,v)/H(u,v)=F(u,v)
When an inverse Fourier transform is performed for F(u,v) to return it to a real plane, a restored image corresponding to the original image f(x,y) can be obtained.
When symbol R is defined as a value obtained by performing the inverse Fourier transform of H−1, similarly, the original image can be obtained by performing a convolution process for the image on the real plane as the following expression.g(x,y)*R(x,y)=f(x,y)
R(x,y) is referred to as an image restoration filter. An actual image contains a noise component. Therefore, when the image restoration filter made by using a complete inverse of the optical transfer function (OTF) as described above is used, the noise component is amplified along with the deteriorated image and generally a good image can not be obtained. In this regard, for example, a method of suppressing a restoration rate at a high frequency side of an image in accordance with intensity rate of an image signal and a noise signal, like a Wiener filter, has been known. The deterioration of the color running component of an image, for example completes to be corrected if a blur for each color component of the image is uniform by the correction of the blur component described above.
Because the optical transfer function (OTF) varies in accordance with a state of the optical system such as a focal distance (zoom state) of the optical system or a stop diameter, the image restoration filter used for the image restoration process also needs to be changed in accordance with the state of the optical system.
Japanese Patent No. 3532368 discloses an endoscope for observing inside a biological object. The endoscope eliminates a blur of an image using a PSF in accordance with a used fluorescence wavelength in a range beyond an in-focus range of an image pickup system. In other words, an objective optical system with a small F-number is necessary because the fluorescence is weak, and an in-focus image is obtained by performing an image restoration process for a range of out of focus because a focal depth is shallow.
As described above, the image restoration process is performed for the input image obtained by taking an image to be able to obtain a restored image of a high quality image in which a blur component has been corrected in accordance with various aberrations of the optical system.
However, in taking an actual image, there may be a possibility that a state of the optical system in obtaining an input image and a state of the optical system corresponding to the image restoration filter which is applied to the input image do not match each other. For example, a case where a three-dimensional object is imaged is considered.
An image pickup apparatus takes an image focusing a focal point on one plane in an object space by an autofocus function or a manual focusing. However, when the object is a three-dimensional object, an object distance is different in accordance with a field angle. In this case, although a part which is in-focus state in the object can be comparatively sharply imaged, a part of out-of-focus state is imaged with a blur in accordance with the distance. When information relating to the object distance obtained in taking an image is only information of an object distance to the in-focus part, the image restoration filter which is optimized for each field angle at the object distance is used.
Therefore, in the image after the image restoration process, because an image restoration filter which is optimized for the in-focus part is applied, a desired sharpness can be obtained in the in-focus part. However, an image restoration filter which is not optimized for the out-of-focus part is applied. Therefore, a blur component remains in the out-of-focus part although a restoration effect is somewhat obtained.
On the other hand, in some cases, the blur in accordance with the object distance is effective for representing a three dimensional appearance in an image or an extracted appearance of a watched object. For example, there is a representing method of intentionally blurring a background by being focused on a main object using a telephoto lens with a shallow depth of field. In this case, even for the image after the image restoration process, it is preferable that the in-focus part (hereinafter, referred to also as an “in-focus object”) is further sharpened and the blur remains in the out-of-focus part (hereinafter, referred to also as an “out-of-focus object”).
However, the inventor has discovered that there is a possibility that a false color may be generated in principle when an image restoration process is performed for the out-of-focus object using an image restoration filter, which is optimized for an in-focus distance but is not optimized for an out-of-focus distance. The false color is, for example generated by increasing the color by the image restoration process when the color is generated in an edge portion of the out-of-focus object in accordance with a chromatic aberration included in the optical system.
Further, in some cases, such a false color may be generated without being limited to taking an image of a three-dimensional object. In other words, the false color is generated when a state of an aberration changed in accordance with the optical system at the time of taking an image and a state of an aberration to which the image restoration filter corresponds are different from each other, whether it is in-focus state or out-of-focus state inherently.
In the image restoration process for improving an image quality, the generation of the false color causes a significant deterioration of the image quality.
Japanese Patent No. 3532368 discloses that a technology for performing an image restoration process for an out-of-focus range to increase a sharpness to improve an image quality. However, because a false color is generated even if the sharpness increases, the image quality in totality can not be improved.